Papers with heuristic and model-based filtering techniques
Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation (2026.eacl-long)
Copied to clipboard
Thomas F Burns, Letitia Parcalabescu, Stephan Waeldchen, Michael Barlow, Gregor Ziegltrum, Volker Stampa, Bastian Harren, Björn Deiseroth
| Challenge: | Recent studies show that data quality can significantly boost performance and training efficiency for large language models. |
| Approach: | They propose a German-language dataset curation pipeline that combines heuristic and model-based filtering techniques with synthetic data generation. |
| Outcome: | The proposed pipeline can be used to create a large-scale German pre-training dataset using common Crawl web data, fineweb2 and synthetically generated data conditioned on real, organic web data. |